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SYSTEM FOR NON-DISRUPTIVE INSERTION AND 
REMOVAL OF NODES IN AN ATM SONET RING 

FIELD OF INVENTION 

The invention relates generally to asynchronous transfer mode (ATM) rings and, 
more particularly, to mechanisms for insertion and removal of ring nodes. 

BACKGROUND OF THE INVENTION 

Communication rings include a plurality of nodes that are interconnected by a 
communication medium, such as fiber optic cable. In known prior ATM rings, the traffic 
on the ring is disrupted when a node is inserted into the ring, that is, when the node is 
brought into the ring operations, and/or when a node is removed from the ring operations. 
In order for the newly inserted node to operate in the ring, the existing nodes must be 
configured to make use of the virtual paths associated with the new node. In one known 
ATM ring, the nodes on the ring must be re-configured individually and manually, that is, 
each must be reprograrnrned. Accordingly, the traffic on the ring is disrupted as the 
operations of each of the respective ring nodes are interrupted so that the node can be 
updated to make use of the new virtual paths. 

The traffic on the ring is also disrupted each time a ring node has a catastrophic 
failure and must be removed from the ring operations. When a node fails, the functioning 
ring nodes must again be reconfigured. In such systems, the traffic on the ring is also 
further disrupted when the failed node is re-booted and re-inserted into the operations of 
the ring. Ring traffic may also be disrupted when a node is temporarily removed from 
the ring for service or upgrading and thereafter re-booted and re-inserted. 

The reconfiguration of the nodes in the prior system must be performed under the 
control of the system manager. Accordingly, the time to reconfigure each node, and thus, 
the time during which traffic on the ring is disrupted at each node, and collectively over 
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the ring, is relatively long. Further, the time it takes to insert a node increases as the 
number of nodes on the ring increases. 

As the rings become larger and more traffic is sent over them, the longer 
disruptions in the traffic are not as easily tolerated. This is particularly the case when the 
5 ring is shared by multiple customers, and any disruption adversely affects more than the 
customer that is, for example, joining the ring. Accordingly, what is needed is a system 
for inserting and removing a node without disrupting the traffic on the ATM ring. 

SUMMARY OF THE INVENTION 

The invention is a system and a method of operating the system for non- 
10 disruptively inserting a node into the operations of an ATM ring by first including the 
node as a virtual path pass through and then using the ring operations to update the 
routing tables at the new node and the existing nodes to include therein the one or more 
virtual paths associated with the new node. The system non-disruptively removes a node 
from the ring essentially by reversing the insertion operations. The system and method 
15 are described below in terms of a SONET ring that handles ATM traffic, but may be used 
in any ATM ring. 

Basically, the node to be inserted establishes communications with a ring hub 
node over an established intra-ring management channel. The node and the ring hub 
node then exchange information over the intra-ring management channel and later over a 

20 virtual path that is assigned to that node, to bring the new node into the ring operations in 
stages without disrupting the flow of ATM traffic over the existing virtual path 
connections. For ease of understanding, we refer hereinafter to the node that is inserted 
into the ring as the "new node" 

More specifically, once the new node is physically connected into the ring it 

25 operates as an optical bypass. The new node then performs its initializing routines and, 
as necessary, communicates with the hub node, to allow the traffic on the existing virtual 
paths to pass through the processor on the new node. The new node next requests that the 
hub node assign one or more virtual paths for directing traffic between the new node and 
the other nodes on the network, that is, between the new node and the other ring nodes 

30 and also the network nodes that are external to the ring. In response to the request, the 
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hub node assigns the virtual path or paths to the new node and establishes a signaling 
channel to the node over one of the virtual paths. The hub node also notifies the other 
nodes of the assignment, and downloads to the new node routing tables that include the 
established connections over the existing virtual paths. 

5 The new node is thus brought fully up-to-date. Accordingly, it may participate in 

traffic shaping on existing virtual circuits, and in call set up and call tear down operations 
over its assigned virtual paths, as discussed in more detail below. 

When a node fails, the system reverses the steps discussed above to remove the 
node from the ring without further disruption of the traffic on the ring. The hub node 

10 learns of the node failure when the failed node no longer communicates with the hub 

node, as discussed in more detail below. The hub node then instructs the operative nodes 
to tear down the virtual paths and associated virtual circuits that originate or end at the 
failed node. The nodes continue to maintain the other virtual path and virtual circuit 
connections over the ring, and the ATM traffic that is directed to the operative nodes is 

15 thus not disrupted. When the failed node is later re-booted, it is brought back into the 
ring operations without disrupting the traffic on the ring using the steps discussed above. 

The operations of the current system are in contrast to known prior systems in 
which the insertion of a new node and/or the failure and re-insertion of a previously 
operative node requires system-operator controlled reconfiguration of the nodes on the 

20 ring. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The invention description below refers to the accompanying drawings, of which: 
Fig. 1 is a functional block diagram of a system constructed in accordance with 
25 the invention; 

Fig. 2 is the system of Fig. 1 with an additional ring node; 
Fig. 3 is a flow chart of the operations of the system of Figs. 1 and 2 to insert a 
node; and 
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Fig. 4 is a flowchart of the operations of the system of Figs. 1 and 2 to removing a 

node.. 

DETAILED DESCRIPTION OF AN ILLUSTRATIVE EMBODIMENT 

Referring now to Fig. 1 , a system 1 0 that may be part of an ATM network 
includes a plurality of nodes 12 u 12 2 , . . . 12i that are referred to collectively as nodes 12. 
One of the nodes, for example, node V2 U operates as a ring hub node to control ring 
operations. 

The nodes 12 are interconnected by a primary ring 14 and a secondary ring 16. 
The exemplary system 10 is a unidirectional SONET ring, with information in the form 
of ATM cells and frames transmitted in both directions over the primary and secondary 
rings. The system 10 may, however, be any type of ATM ring. We discuss below how a 
new node is non-disruptively inserted into the ring and how a failed node is non- 
disraptively removed from the ring. First, however, we discuss in general the operations 
of the system 10. 

1. Ring Operations In General 

Each node 12 receives information over the rings 14 and 16 through incoming 
ports 20 and sends information over the rings through destination ports 22. The nodes 12 
also connect to other systems (not shown), including other networks or rings, or to 
various user terminals, through other incoming and destination ports 24 and 26. 

The ATM cells are sent over the rings from source nodes to destination nodes in 
virtual circuits. The system groups the virtual circuits into virtual paths that include other 
virtual circuits with the same source and destination nodes. 

Each virtual path is identified by a virtual path index (VPI), and each virtual 
circuit is similarly identified by a virtual circuit index (V CI). The VPI and VCI values 
are included in ATM cell headers, and are used by successive nodes to determine how to 
route the cells. 

Intermediate nodes manage the cells on a virtual path basis. When an 
intermediate node receives cells over a virtual path on the primary ring, the node 
forwards the cells to succeeding nodes on the primary ring over the same virtual path. 
The intermediate node may, however, use the virtual circuit information for traffic 
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shaping in the virtual path, in a conventional manner. Similarly, when an intermediate 
node receives cells over a virtual path on the secondary ring the node forwards the cells 
to succeeding nodes on the secondary ring over the same virtual path, and uses the virtual 
circuit information for traffic shaping. 

Each destination node maintains routing tables (not shown) that reference at least 
the associated virtual circuits over both the primary and secondary rings and the 
destination ports to which the virtual circuits are directed. The destination nodes select 
one ring interface, based on the quality of the traffic received over both rings, and direct 
the traffic from the selected interface to the appropriate ports. 

The nodes 12 exchange ring management information with the hub node 12\ over 
a dedicated intra-ring management channel. In particular, the hub node 12\ and the other 
ring nodes 12 exchange ring topology information over the intra-ring management 
channel. In the exemplary system 10, the topology information includes updates to 
virtual path and virtual circuit routing tables (not shown) that are maintained at each of 
the nodes, and also information relating to the responsiveness or non-responsiveness of 
the various nodes. The ring nodes 12 also provide to the hub node 12 X certain other 
information concerning the integrity of the primary and secondary rings by, for example, 
periodically sending OAM cells to the hub node over the intra-ring management channel. 

2. Ring Operations For The Insertion Of A Node 

Referring now also to Fig. 2, before a new nodel2j is inserted into the system 10, 
that is, before the node participates in ring operations, the cabling and the node hardware 
are installed in a conventional manner. Accordingly, to install a new node 12j between, 
for example, the nodes 12 3 and 12 4 , the cabling 15 for the primary ring 14 and the cabling 
17 for the secondary ring 16 that connect nodes 12 3 and 12 4 are unplugged from the ports 
20 and 22 of the node 12 4 . While the cabling is unplugged, the traffic between the nodes 
12 3 and 12 4 is re-routed around the ring. The unplugged cabling is then plugged into the 
appropriate incoming and destination ports of the new node 12j and additional cabling is 
run between the new node and the node 12 4 . Electrical connections for the two rings are 
thus made through the connection hardware 13j of the new node 12j. The added cabling 
is depicted in the drawing by dotted lines. 
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Referring as well to Fig. 3, once the cabling is in place, the new node 12j operates 
as an optical bypass, to physically pass ring traffic between the nodes 12 3 and 12 4 (step 
300). The new node 12j must then be inserted into the ring operations, so that users 
connected to the ring through the node 12j can send and receive information over the 
ring. 

The new node 12j performs its initialization routines, places all virtual paths in 
pass through mode, and communicates with the hub node 12i over the intra-ring 
management channel to exchange information that allows a node processor in the new 
node 12j to control traffic passing through the new node. In particular, the hub node and 
the new node 12j exchange information that allows the node processor to process the 
ATM traffic on the existing assigned virtual paths through the ports 20j and 22j (steps 
302, 304, 306). In the example, the new node contains a list of a default range of virtual 
paths from which all of the virtual paths on the ring are assigned, and the new node 
essentially configures itself, without hub node control, to operate as a virtual path pass 
through. If the new node did not contain such a list, the hub node would instead provide 
a list of the assigned virtual paths. 

The new node 12j next sends a request to the hub node 12i over the intra-ring 
management channel, asking the hub node to assign to the new node one or more virtual 
paths. The new node will later use these virtual paths to establish connections over 
virtual circuits to the other nodes in the system, both on and external to the ring (step 
308). With the new node requesting its virtual path assignment, the hub node is freed 
from monitoring the passive activities of this node and the other new nodes that are being 
brought into the ring operations. Accordingly, the hub node may instead manage the 
dynamic operations on the ring, such as call set up and tear down. 

In response to the request from the new node, the hub node assigns one or more 
virtual paths to the node. The hub node may then notify the other nodes of the new 
assignment (step 310). Alternatively, if the path assignment is within the default range, 
the hub node may instead update the topology information to include the new node and 
rely on call set up operations to update the routing tables when connections are 
established over the newly assigned virtual paths, as discussed below. The ring nodes 12 
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may instead learn of the new node through cells that are periodically introduced to test 
ring continuity and/or to verify ring topology. 

The hub node next establishes a signaling channel over one of the virtual paths 
assigned to the new node. The hub node and the new node can then exchange call set up 
and tear down information over the signaling channel (step 312). The hub node next 
downloads to the new node 12j over the management channel a set of up-to-date routing 
tables that include all of the existing virtual path assignments and/or the associated virtual 
circuit connections (step 314). The routing tables are preferably the tables discussed in 
co-pending United States patent application Serial No. 09/344,845 entitled RAPID CALL 
ESTABLISHMENT IN ATM RINGS which is assigned to a common assignee and is 
incorporated herein by reference. In the exemplary system, the hub node also downloads 
an error checking code, such as a check sum, that is used by the new node 12j to verify 
that the routing tables were downloaded without error. 

The node 12j may now fully participate in the ring operations. More specifically, 
the node 12j sends over the established signaling channel requests to the hub node 12i to 
set up calls from associated users as virtual circuits over a newly assigned virtual path, 
and later to tear down inactive connections over the assigned virtual path. This call set 
up and tear down information is also sent by the hub node to the other nodes on the ring, 
and these nodes then update their routing tables to include the connections. Based on 
instructions and update information received by the new node at various times from the 
hub node, the node 12j updates its routing tables as new connections are established 
between the various other ring nodes, and as various connections are torn down. Also, 
the new node uses the virtual circuit information in the routing tables to shape traffic over 
the various virtual paths in a conventional manner. 

3. Ring Operations For Removal Of A Failed Node 

Referring now to Fig. 4, when a node 12 fails, the system 10 removes the failed 
node from the ring operations, without disrupting the operations of the remaining ring 
nodes, essentially by reversing the steps for the insertion of a node. The hub node learns 
of the failure of, for example, node 12 3 , when the node 12 3 stops communicating with the 
hub node over the signaling channel established between them. The hub node may also 

7 

H:\104\141\0004\PROSECUT\PATAPP.doc 



PATENT 
104141-0004 

learn of the failure of the node 12 3 when the hub node does not, during a predetermined 
maximum time interval, receive OAM cells or other cells that are periodically originated 
by the node 12 3 . Alternatively, the hub node may learn of the failure when the node does 
not respond appropriately to the OAM cells or other test cells that the hub node originates 
and sends over the ring to test ring continuity and/or to verify ring topology (step 400). 

Once the hub node has determined that the node 12 3 has failed, the hub node 
instructs the other nodes on the ring to, as necessary, re-route traffic through the operative 
nodes as part of a protection switching operation (step 402). If the electrical connections, 
that is, the cables and associated hardware, through the failed node are functional, the 
failed node 12 3 operates by default as an optical by-pass and the ring traffic need not be 
re-routed. 

The hub next instructs the operative ring nodes to tear down the virtual circuits 
that originate from or are directed to the failed node (step 404). In response, the nodes 
update their routing tables. The hub node then sends updated ring topology information 
to the remaining nodes, and instructs them to remove from their routing tables the virtual 
paths associated with the failed node. At the same time, the remaining ring nodes 12 
continue to send and receive ATM traffic associated with the operative ring nodes over 
the ring in the usual manner, that is, over the existing virtual paths and associated virtual 
circuits. 

When the failed node 12 is later replaced or re-booted, the new or newly operative 
node is inserted into the ring operations in accordance with the steps discussed above 
with reference to Fig. 3, and the traffic over the ring is not disrupted. The same node 
insertion operations may also be followed after a node has been taken temporarily out of 
ring operations for upgrading and/or service. 

The system and method of operating the system described above allow ring traffic 
to travel the ring without disruption while nodes are added or failed nodes are removed 
from the ring. This is in contrast to the operations of known prior ATM rings that require 
the nodes to be individually reconfigured under the control of a system manager, in order 
to add or remove a node from ring operations. 

The foregoing description has been limited to a specific embodiment of this 
invention. It will be apparent, however, that variations and modifications may be made 
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to the invention, such as, for example, use of any type of communication media between 
nodes, configuration of the ring as any type of dual-path communication network, with 
the attainment of some or all of its advantages. Therefore, it is the object of the appended 
claims to cover all such variations and modifications as come within the true spirit and 
scope of the invention. 

What is claimed is: 
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1 . A method for inserting a given node into ring operations of an ATM ring, 
including the steps of: 

A. operating the given node as a bypass for ATM traffic on the ring; 

B. operating the given node to pass through virtual paths known to the given 

node; 

C. establishing connection to inter-ring management channel; 

D. exchanging information between the given node and a ring hub node to operate 
the given node as a pass through for the ATM traffic on other existing virtual path 

connections on the ring; 

E. assigning to the given node one or more virtual paths to direct traffic to and 

from the given node over the ring; 

F. communicating the virtual path assignment to all nodes on the ring; and 

G. providing to the given node connection information for all of the virtual paths 
and virtual circuits on the ring. 

2. The method of claim 1 wherein the step of providing connection information to the 
given node includes providing routing tables to the given node. 

3. The method of claim 2 wherein the step of providing connection information further 
includes providing the information from the hub node to the given node. 

4. The method of claim 3 wherein the step of providing connection information further 
includes 

i. providing an error checking code with the information, and 

ii. at the given node checking the information with the error checking code to 
determine that the information is correct. 

5. The method of claim 1 wherein the step of assigning the virtual path to the given node 
further includes 
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i. the given node requesting the assignment from the hub node, and 

ii. the hub node responding to the request with the assignment. 

6. The method of claim 1 wherein the step of communicating the virtual path assignment 
to all of the nodes includes updating routing tables maintained by the nodes. 

7. The method of claim 1 wherein the step of communicating the virtual path assignment 
to all of the nodes includes providing to the nodes call set up information for calls over 
the newly assigned virtual path. 

8. The method of claim 1 further including the steps of 

F. establishing connections to and from the given node over the assigned virtual 
path; and 

G. tearing down connections over the assigned virtual path. 

9. The method of claim 8 wherein the step of communicating the virtual path assignment 
to all of the nodes includes updating routing tables maintained by the nodes. 

10. The method of claim 9 further including updating the routing tables with call set up 
and tear down information associated with the one or more virtual paths assigned to the 
given node. 

1 1 . The method of claim 1 further including the step of 5 at the given node, shaping traffic 
over the virtual circuits associated with the established connections on the ring. 

12. A method for removing a failed node from an ATM ring, the method including the 
steps of : 

A. determining that a given node has failed; 

B. tearing down virtual circuit connections directed to or initiating from the failed 

node; 

C. tearing down virtual paths assigned to the failed node; and 
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D. providing to the remaining nodes on the ring updated ring topology 
information that excludes the failed node. 

13. The method of claim 12 wherein 

I the step of determining that the given node has failed includes having a ring 
hub node determine the failure, and 

ii. the steps of tearing down the virtual circuit and virtual path connections are 
controlled by the hub node. 

14. A method for inserting a given node into ring operations of an ATM ring and 
removing a failed node from the ring operations, the method including the steps of: 

A. operating the given node as a bypass for ATM traffic on the ring; 

B. placing all other known virtual paths in by pass mode; 

C. establishing connection to inter-ring management channel; 

D. exchanging information between the given node and a ring hub node to operate 
the given node as a pass through for the ATM traffic on existing connections on the ring; 

E. assigning to the given node one or more virtual paths to direct traffic to and 
from the given node over the ring; 

F. communicating the virtual path assignment to all nodes on the ring; and 

G. providing to the given node connection information for all of the virtual paths 
and virtual circuits on the ring. 

15. The method of claim 14 wherein the step of providing connection information to the 
given node includes providing routing tables to the given node. 

16 The method of claim 15 wherein the step of providing connection information further 
includes providing the information from the hub node to the given node. 

17. The method of claim 15 wherein the step of providing connection information further 
includes 

iii. providing a error checking code with the information, and 
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iv. at the given node checking the information with the error checking code to 
determine that the information is correct. 

18. The method of claim 14 wherein the step of assigning the virtual path to the given 
node further includes 

iii. the given node requesting the assignment from the hub node, and 

iv. the hub node responding to the request with the assignment. 

19. The method of claim 14 wherein the step of communicating the virtual path 
assignment to all of the nodes includes updating routing tables maintained by the nodes. 
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ABSTRACT OF THE DISCLOSURE 

A system and a method of operating the system for non-disruptively inserting a 
node into the operations of an ATM ring uses the ring operations to update topology 
information and routing tables at the existing nodes to include therein the new node and 
one or more virtual paths associated with the new node. Basically, the node to be 
inserted establishes communications with a ring hub node over an established intra-ring 
management channel. The node and the ring hub node then exchange information over 
the management channel and the node performs its initializing routines in order that the 
node may operate to pass through traffic on the existing virtual paths. The new node next 
requests from the hub node the assignment of one or more virtual paths for directing 
traffic between that node and other nodes on the network. In response to the request, the 
hub node assigns one or more virtual paths to the requesting node and establishes a 
signaling channel to the node over one of the virtual paths. The hub node also notifies 
the other nodes of the assignment, and downloads to the new node routing tables that 
include the established connections over the existing virtual paths. The node may then 
participate in traffic shaping on existing virtual circuits, and also in call set up and call 
tear down operations over its assigned virtual paths in the same manner as the other 
nodes on the ring. A failed, or otherwise inoperative, node is removed from the ring 
without disrupting the traffic on the ring by essentially reversing the node insertion 
operations. After learning of the node failure, the hub node instructs the operative nodes 
to tear down the virtual path and associated virtual circuits that originate from or end at 
the failed node. Thereafter, the hub node directs the nodes to update their ring topology 
information to remove the failed node. In the meantime, the nodes continue to maintain 
the other virtual path and virtual circuit connections over the ring without disruption. 
When the failed node is later re-booted, the system follows the steps discussed above to 
re-insert the node, without disrupting the traffic on the ring. 
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NODE 12j OPERATES AS AN OPTICAL 
BYPASS 



THE NODE 12j OPERATES TO PASS THE EXISTING 
VIRTUAL PATHS THROUGH THE NODE INTERFACES 



NODE 12j ESTABLISHES CONNECTION TO WELL 
KNOWN INTRA-RING MANAGEMENT CHANNEL 
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306 



NODE 12 AND HUB NODE 12 1 COMMUNICATE OVER THE 
ESTABLISHED INTRA-RING MANAGEMENT CHANNEL 



IN RESPONSE TO A REQUEST FROM THE NODE 12 THE HUB NODE 
ASSIGNS TO THE NODE ONE OR MORE VIRTUAL PATHS FOR 
SENDING TRAFFIC TO THE OTHER NODES IN THE NETWORK 



f 



308 



THE HUB SENDS TO THE OTHER NODES RING TOPOLOGY 
UPDATES THAT INCLUDE NEWLY ASSIGNED VIRTUAL PATHS 



THE HUB NODE ESTABLISHES A SIGNALLING CHANNEL 
WITH THE NODE 12 : ON AN ASSIGNED VIRTUAL PATH 



THE HUB NODE DOWNLOADS VIRTUAL PATH AND VIRTUAL 
CIRCUIT TABLES TO NODE 12, 



NODE 12j PARTICIPATES FULLY IN RING OPERATIONS 
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314 



316 



FIG. 3 



THE HUB NODE DETERMINES A NODE 
HAS FAILED BY A LACK OF 
COMMUNICATION FROM THE NODE 



400 



PROTECTION SWITCHING IS 
INITIATED AS NEEDED 



402 



THE REMAINING NODES ARE 
INSTRUCTED TO TEAR DOWN THE 
VIRTUAL PATHS AND VIRTUAL CIRCUITS 
ASSOCIATED WITH THE FAILED NODE 



404 



406 



THE REMAINING NODES ARE INSTRUCTED TO 
UPDATE THEIR RING TOPOLOGY INFORMATION 
AND ROUTING TABLES TO REMOVE THE FAILED 

NODE 



FIG. 4 
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